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TITLE OF THE INVENTION 
IMAGE PROCESSING APPARATUS AND METHOD, AND COMPUTER- 
READABLE MEMORY 



FIELD OF THE INVENTION 
This invention relates to an image processing 
apparatus and method for processing a moving picture, 
and to a computer-readable meraorY storing the program 
code for this processing. 

EACKGROUND OF THE INVENTION 
When only a specific scene (a part of a TV program 
that begins from a specific title screen of the program, 
or a scene in which a specific person appears) is to be 
extracted from a plurality of moving-picture materials 
gathered by a video camera or VTR, etc., the user 
searches for the desired scene by repeatedly fast- 
forwarding or playing back each moving-picture material, 
writes down the time code of each scene and creates a 
list of time codes. Then, on the basis of the created 
list of time codes, the user performs an editing 
operation such eis gathering specific scenes together and 
combining them into one moving picture . 

However, in order to extract a specific scene from 
a plurality of moving -picture materials, the operator 
must perform the manual operations of f ast-f oirTvard and 



playback, search for the desired scene and create the 
list of time codes. This requires a great deal of time 
and labor. 

5 SUMMARY OF THE INVENTION 

Accordingly, an object of the present invention is 
to provide an image processing apparatus, method and 
computer- readable memory which make it possible to 

10 extract a desir€;d scene from a plurality of moving- 
picture materials in an efficient manner . 

According to the present invention, the foregoing 
object is attained by providing an image processing 
apparatus for processing a moving picture, comprising: 

15 frame extraction means for extracting frames 

constituting an entered moving picture; discrimination 
means for discriminating a scene change by comparing 
frames extracted by the frame extraction means; storage 
means for storing scene-change information relating to 

20 the scene change; discriminated by the discrimination 

means; designating means for designating an image that 
corresponds to a scene that is the object of a search; 
comparison means for comparing a scene-change frame, 
which is obtained by referring to the scene-change 

25 information that has been stored in the storage means, 
and the image that has been designated by the 
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designation means; and scene extraction means for 
extracting a sce.n& that corresponds to the image based 
upon result of the comparison performed by the 
comparison means. 
5 The foregoing object is attained by providing an 

image processing method for processing a moving picture, 
comprising: a frame extraction step of extracting 
frames constituting an entered moving picture; a 
discrimination step of discriminating a scene change by 

10 comparing frames extracted at the frame extraction step; 
a storage step of storing scene-change information 
relating to the scene change discriminated at the 
discrimination step; a designating step of designating 
an image that corresponds to a scene that is the object 

15 of a search; a comparison step of comparing a scene- 
change frame, which is obtained by referring to the 
scene-change information that has been stored at the 
storage step, and the image that has been designated at 
the designation step; and a scene extraction step of 

20 extracting a scene that corresponds to the image based 
upon result of the comparison performed at the 
comparison step.. 

The foregoing object is attained by providing a 
computer-readable memory storing program code of image 

25 processing for processing a moving picture, the memory 
including: program code of a frame extraction step of 
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extracting frames constituting an entered moving 
picture; program code of a discrimination step of 
discriminating a. scene change by comparing frames 
extracted at the frame extraction step; program code of 
5 a storage step of storing scene-change information 
relating to the scene change discriminated at the 
discrimination step; program code of a designating step 
of designating an image that corresponds to a scene that 
is the object of a search; program code of a comparison 

10 step of comparing a scene-change frame, which is 

obtained by referring to the scene-change information 
that has been stored at the storage step, and the image 
that has been designated at the designation step; and 
program code of a scene extraction step of extracting a 

15 scene that corresponds to the image based upon result of 
the comparison performed at the comparison step. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 

20 drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 

Fig, 1 is a block diagram illustrating the 
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construction of an image processing apparatus according 
to an embodiment of the present inventions- 
Fig. 2 is ci block diagram illustrating the 
functional construction of an image processing apparatus 
5 according to an embodiment of the present invention; 

Fig. 3 is a flowchart illustrating the procedure of 
scene-change detection processing according to an 
embodiment of the present inventions- 
Fig. 4 is diagram illustrating an example of block 
10 division of a frame according to an embodiment of the 
present invention; 

Fig. 5 is diagram illustrating an example of scene- 
change information according to an embodiment of the 
present invention; 
15 Fig. 6 is a flowchart illustrating the procedure of 

scene extraction processing according to an embodiment 
of the present invention; and 

Fig. 7 is a diagram showing an example of a list of 
extracted scenes according to an embodiment of the 
20 present invention. 



DESCRIPTION OF THE PREFERRED EMBODIMENT 



25 



A preferred embodiment of the present invention 
will now be described in derail with reference to the 
accompanying drawings . 



Fig. 1 is cL block diagram illustrating the 
construction of an image processing apparatus according 
to a preferred embodiment of the present invention. 

As shown in Fig. 1, the apparatus includes a CPU 
5 101 for implementing various control operations 

regarding various components of the image processing 
apparatus; a ROM 102 for storing a booting program, 
which is run when the image processing apparatus is 
started up, and various data; a RAM 103 which stores a 
10 control program run by the CU 101 and which provides a 
working area when various control operations are 
executed; a keyboard 104 and a mouse 105 for providing 
an environment that allows a user to make various 
inputs; an external storage device 106 constituted by a 
15 hard disk, floppy disk or CD-ROM; a display unit 107 

constituted by a CRT or LCD or the like which displays 
the results of processing for the user; a network 
interface controller (NIC) 108 for making possible 
communication with various devices on a network; a video 
20 interface (I/F) 109 that makes it possible to capture a 
moving picture from a video camera 110 or VTR 112; and a 
bus 111 for interconnecting the above-mentioned 
components . 

It should be noted that the video camera 110, VTR 
25 112 and external storage device 106 may be replaced by 
components connected to the network. 
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The functions of the image processing apparatus 
according to this embodiment will be described with 
reference to Fig. 2. 

A moving-picture input unit 2 01 is for inputting a 

5 moving picture captured from the video camera 110 or VTR 
112 via the video interface 109 or a moving picture 
downloaded from the network via the NIC 108. A frame 
extraction unit 202 successively extracts frames 
(images) , which constitute the entered moving picture, 

10 in regular ordei:. A frame storage unit 209 stores 

frames extracted by the frame extraction unit 2 02. A 
scene-change discrimination unit 2 03 examines each frame 
to discriminate a scene change in a moving picture using 
an interframe- similarity calculation unit 204. The 

15 latter calculates the degree of similarity between 

frames. A scene-change information storage unit 205 
stores information relating to a scene changes that has 
been discriminaLed by the scene-change discrimination 
unit 203. A pattern image designation unit 206 allows 

2 0 the user to designate a pattern image using the keyboard 
104 and mouse 105. A scene extraction unit 207 uses the 
inter frame -similarity calculation unit 204 to compare 
the pattern image that has been designated by the 
pattern image designation unit 2 06 with a frame, in 

25 which a scene change has occurred, obtained by referring 
to the scene-change information storage unit 205, and 



extracts a desired scene corresponding to the pattern 
image. An extra cted-scene list display unit 208 causes 
the display unit 107 to display a list of information 
relating to scenes extracted by the scene extraction 
unit 207. 

An example of operation of the image processing 
apparatus havincf the scene extraction function of this 
embodiment arranged as set forth above will now be 
described. 

[Scene-change detection processing] 

Processing for detecting a change in scene will be 
described with reference to Fig. 3. 

Fig. 3 is a flowchart illustrating the procedure of 
scene-change detection processing according to this 
embodiment of the present invention. 

First, a moving picture to undergo processing is 
input by the moving-picture input unit 201 at step S301. 
Next, at step S302, the frame extraction unit 202 
executes processing for extracting one frame at a time 
from the entered moving picture. Then, at step S303, 
the frame extraction unit 202 stores the extracted 
frames in the frame storage unit 209. 

It is determined at step S3 04 whether all frames of 
the entered moving picture have been extracted. If all 
frames have not been extracted ("NO" at step S304), 
steps S302 and S303 are repeated. If all frames have 



been extracted ("YES" at step S304) , then control 
proceeds to step S305. 

Step S3 05 calls for the scene-change discrimination 
unit 203 to use the interf rame-similarity calculation 
5 unit 204 to perforin, in numerical order, an interframe- 
similarity calculation between a frame of interest and a 
group of past frames. There are instances where the 
group of past frames is a single frame or a plurality of 
frames, and there is no particular limitation upon the 

10 algorithm. 

The simplest example will now be described. First, 
the immediately preceding frame and the current frame 
are each divided into a plurality of blocks vertically 
and horizontally, as shown in Fig. 4. Next, average 

15 values of R, G amd B are calculated in regard to each 
block. This is followed by obtaining the sum of the 
squares of the differences between the corresponding R, 
G and B channels in the preceding and current frames . 
This sum of the squares is taken as inter frame 

20 similarity, which represents the degree of intensity of 
the scene change; . The smaller the value of the sum of 
the squares, the greater the similarity between frames; 
the larger the value, the smaller the similarity. In 
other words, this is a method of obtaining an instance 

2 5 in which the possibility of a scene change is high. A 
example of the calculation formula is as follows: 



X [(PliR-P2iR)2 + (PliG-P2iG)2 + (PliB-P2iB)2] 
i=l 

where i represents the block undergoing processing, and 
K : nizmljer of blocks into which the frame is 
divided; 

Pli^: average value of R channel of ith block of 

preceding frame ; 
PliG: average value of G channel of ith block of 

preceding frame ; 
Plie: average value of B channel of ith block of 

preceding frame; 
P2iR: average value of R channel of ith block of 

current frame ; 
P2iG: average value of G channel of ith block of 

current frame; 
P2iB: average value of B channel of ith block of 

current frame 
On the basis of the result of calculation of 
interframe similarity, the scene-change discrimination 
unit 2 03 determines whether or not there has been a 
scene change. If there has not been a scene change 
("NO" at step S306) , control proceeds to step S308. If 
there has been a scene change ("YES" at step S306) , on 
the other hand, then, at step S307, the ID of the 
entered moving picture, the ID of the scene change and 
the frame are stored as the scene starting frame and the 
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similarity (or distance) representing the degree of 
intensity of the scene change is stored. These items of 
infoirmation shall be referred to as "scene-change 
information" . An example of scene-change information is 
5 illustrated in Fig. 5. This information is stored in 
the scene-change information storage unit 205. 

Next, at step S3 08, it is determined whether all 
frames of the moving picture have been discriminated for 
scene change. If all frames have not undergone scene- 
10 change discrimination ("NO" at step S308), processing 

from step S3 05 to step S3 07 is repeated until all scene- 
change frames of the moving picture and the similarity 
information are obtained. If all frames have undergone 
scene-change discrimination ("YES" at step S308), on the 
15 other hand, then processing is exited. 

The foregoing is processing for detecting scene 
changes and is cipplied to all moving pictures that are 
to undergo scene extraction. 

[Scene extraction processing] 
2 0 Reference will be had to Fig. 6 to describe 

processing for designating a pattern image and 
extracting a desired scene. 

Fig. 6 is a flowchart illustrating the procedure of 
scene extraction processing according to this embodiment 
2 5 of the present invention. 

The user designates a pattern image using the 
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pattern image designation unit 206 at step S501. The 
pattern image designated is the image that represents 
the lead frame of a scene (e.g., the title screen of a 
specific part of a TV program) . One conceivable method 
5 of designating the pattern image is for the user to 
create an image that resembles the lead frame of the 
scene. Next, at step S502, the scene extraction unit 
207 regards the pattern image, which has been designated 
by the pattern image designation unit 205, as being a 

10 frame, and uses the interframe- similarity calculation 
unit 204 to calculate the degree of similarity between 
this frame and the already detected scene-change frame. 
Though the method of calculating similarity is not 
particularly limited, one conceivable method is that 

15 mentioned in the scene-change detection processing 

described above.. The scene extraction unit 2 07 extracts 
the scene-change frame from the frame storage unit 209 
upon referring to the ID of the moving picture and the 
starting frame of the scene-change information in the 

20 scene-change information storage unit 205. 

Next, at step S503, the scene extraction unit 2 07 
determines whether the pattern image is a frame 
identical with the scene- change frame based upon the 
degree of similarity calculated by the interframe- 

25 similarity calculation unit 204. If the frames are not 
identical ("NO" at step S503), control proceeds to step 
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S505. If the frames are identical ("YES" at step S503), 
on the other hand, then, at step S504, the ID of the 
moving picture is output to the extracted- scene list 
display unit 2 08 as well as the scene-change frame, 
5 which serves as the starting frame of a scene, and the 
frame (which will serve as the end frame of the scene) 
that immediately precedes the next scene-change frame. 
An example of a list of the extracted scenes is shown in 
Fig. 7. 

10 Next, at step S505, it is determined whether all 

scene-change frames have been compared with the pattern 
image. If all scene-change frames have not been 
compared with the pattern image ("NO" at step S505), 
steps S502 to S!304 are repeated to extract scenes that 

15 begin with those frames of all scene-change frames that 
are identical with the pattern image. If all scene- 
change frames have been compared with the pattern image 
("YES" at step S505) , then control proceeds to step 
S506. This is followed by step S506, at which the 

20 extracted-scene list display unit 2 08 causes the list of 
extracted scenes (Fig. 7) to be displayed for the user. 
This concludes scene extraction processing. 
Thus, in accordance with this embodiment, as 
described above, scene changes in an input moving 

25 picture can be detected automatically and the user, 

merely by designating a pattern image, can automatically 
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extract a frame of a scene, which corresponds to the 
pattern image. As a result, the user need not perforin 
such operations as detection of scenes and creation of a 
time-code list, thus making it possible to perform 
5 operations such as editing in an efficient manner. 
[Other embodiments] 

In the embodiment described above, all frames are 
extracted and stored through steps S3 02 to S3 04, after 
which discrimincLtion of a change in scene is performed 

10 at steps S305 to S3 08. However, it is permissible to 
adopt an arrangement in which discrimination of scene 
change is perfonmed successively whenever a frame is 
extracted, and only the necessary frames (the scene- 
change frames, etc.) are stored in memory. 

15 Further, though the list of extracted scenes (Fig. 

7) per se is displayed for the user, it may be so 
arranged that the extracted scenes are edited into a 
single moving picture based upon the scene list and the 
scene list is displayed for the user. Alternatively, a 

20 printer may be connected to the apparatus to output the 
moving picture on a recording medium. 

Further, a method in which the user creates the 
pattern image has been described as the method of 
designating the pattern image. However, it is 

25 permissible to adopt an arrangement in which scene- 
change frames extracted in scene-change detection are 
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displayed for the user and the user selects a frame from 
these selected frames as the pattern image. It may also 
be so arranged that the selection is made from a group 
of other images that have been prepared in advance . 
5 Further, though only one scene that starts from a 

frame resemblincf the pattern image is extracted in the 
foregoing embodiment, it may be so arranged that the 
user can designeite the number of scenes, thereby making 
it possible to extract a designated number of scenes 

10 that start with a frame resembling the pattern image. 

It is permissible to designate a number of frames with 
regard to frames prior to and with regard to frames on 
and after a fraii:ie resembling the pattern image. 

Further, though only one scene that starts from a 

15 frame resembling the pattern image is extracted in the 
foregoing embodiment, it may be so arranged that the 
user can designate scene length (time) , thereby making 
it possible to deduce time from the niomber of frames of 
a scene based upon scene-change information and extract 

20 a scene of a designated length that begins with the 

frame resembling the pattern image. It is permissible 
to designate scene length (time) with regard to frames 
prior to and with regard to frames on and after a frame 
resembling the pattern image. 

25 Only one scene that starts from a frame resembling 

the pattern image is extracted in the foregoing 
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embodiment. However, if the frame immediately preceding 
a scene-change frame is extracted from the scene-change 
information and compared with the pattern image at step 
S502, then it will be possible to extract a scene that 
5 ends with a frame resembling the pattern image. 

Again, only one scene that starts from a frame 
resembling the pattern image is extracted in the 
foregoing embodiment. However, if the pattern image is 
compared with all frames, a similar frame is found and a 

10 scene in which this frame is present is obtained by 

referring to the scene-change information, then it will 
be possible to extract a scene containing a frame 
resembling the pattern image. 

As described above, this embodiment enables user to 

15 automatically extract a frame, which corresponds to the 
pattern image, that is the leading, intermediate or 
final frame of a scene. In addition, extracted scenes 
can be combined into a single moving picture 
automatically. 

20 The present invention can be applied to a system 

constituted by a plurality of devices (e.g., a host 
computer, interface, reader, printer, etc.) or to an 
apparatus comprising a single device (e.g., a copier or 
facsimile machine, etc.) . 

25 Furthermore, it goes without saying that the object 

of the invention is attained by supplying a storage 
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mediim storing the program codes of the software for 
performing the functions of the foregoing embodiments to 
a system or an apparatus, reading the program codes with 
a computer (e.g., a CPU or MPU) of the system or 
5 apparatus from the storage medium, and then executing 
the program codes . 

In this case, the program codes read from the 
storage medium implement the novel functions of the 
invention, and the storage medium storing the program 
10 codes constitutes the invention. 

Further, the storage medium, such as a floppy disk, 
hard disk, optical disk, magneto-optical disk, CD-ROM, 
CD-R, magnetic tape, non-volatile type memory card or 
ROM can be used to provide the program codes . 
15 Furthermore, besides the case where the aforesaid 

functions according to the embodiments are implemented 
by executing the program codes read by a computer, it 
goes without saying that the present invention covers a 
case where an operating system or the like running on 
20 the computer performs a part of or the entire process in 
accordance with the designation of program codes and 
implements the functions according to the embodiments. 

It goes without saying that the present invention 
further covers a case where, after the program codes 
25 read from the storage medium are written in a function 
expansion board inserted into the computer or in a 
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memory provided in a function expansion unit connected 
to the computer, a CPU or the like contained in the 
function expansion board or function extension unit 
performs a part of or the entire process in accordance 
with the designation of program codes and implements the 
function of the above embodiment. 

In a case where the present invention is applied to 
the above-mentioned storage medium, program code 
corresponding to the flowcharts of Figs . 3 and 6 
described earlier would be stored on the storage medium. 

Thus, in accordance with the embodiments of the 
present invention as described above, it is possible to 
provide an image processing apparatus, method and 
computer-readable memory through which a desired scene 
can be extracted from a plurality of moving-picture 
materials in an efficient manner. 

As many apparently widely different embodiments of 
the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific 
embodiments thereof except as defined in the appended 
claims . 
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WHAT IS CLAIMED IS: 

1 . An image processing apparatus for processing a 
moving picture, comprising: 

frame extraction means for extracting frames 
5 constituting an entered moving picture; 

discrimination means for discriminating a scene 
change by comparing frames extracted by said frame 
extraction means; 

storage means for storing scene-change information 
10 relating to the scene change discriminated by said 
discrimination means; 

designating means for designating an image that 
corresponds to a scene that is the object of a search; 
comparison means for comparing a scene-change 
15 frame, which is obtained by referring to the scene- 
change information that has been stored in said storage 
means, and the image that has been designated by said 
designation means; and 

scene extraction means for extracting a scene that 
20 corresponds to the image based upon result of the 
comparison performed by said comparison means. 

2. The apparatus according to claim 1, wherein said 
comparison means includes computation means for 
computing degree of similarity between the scene-change 

25 frame and the image that has been designated by said 
designation means; 
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wherein said scene extraction means extracts the 
scene corresponding to said image based upon results of 
calculation performed by said calculation means . 

3. The apparatus according to claim 1, further 
comprising output means for outputting scene information 
relating to scenes that have been extracted by said 
scene extraction means . 

4. The apparatus according to claim 3, wherein said 
output means edits scenes that have been extracted by 
said scene extraction means and combines these extracted 
scenes into a single moving picture. 

5. The apparatus according to claim 1, wherein said 
designating means designates a pattern image that 
corresponds to any of a leading, intermediate or final 
frame of a scene that is the object of a search. 

6. The apparatus according to claim 1, wherein said 
designating means is capable of designating a number of 
scenes to be ej<tracted. 

7. The apparatus according to claim 1, wherein said 
designating means is capable of designating the time of 
a scene to be extracted. 

8. The apparatus according to claim 1, wherein said 
designating meems is capable of designating a number of 
scenes to be extracted, with regard to frames prior to 
and with regard to frames on and after a frame 
corresponding to the pattern image. 
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9. The apparatus according to claim 1, wherein said 
designating means is capable of designating time of a 
scene to be extracted, with regard to frames prior to 
and with regard to frames on and after a frame 
corresponding to the pattern image. 

10. An image processing method for processing a moving 
picture, comprising: 

a frame extraction step of extracting frames 
constituting an entered moving picture; 

a discrimination step of discriminating a scene 
change by comparing frames extracted at said frame 
extraction step; 

a storage step of storing scene-change information 
relating to the scene change discriminated at said 
discrimination step; 

a designating step of designating an image that 
corresponds to a scene that is the object of a search; 

a comparison step of comparing a scene-change 
frame, which is obtained by referring to the scene- 
change information that has been stored at said storage 
step, and the image that has been designated at said 
designation step; and 

a scene extraction step of extracting a scene that 
corresponds to the image based upon result of the 
comparison performed at said comparison step. 
11. The method according to claim 10, wherein said 



comparison step includes a computation step of computing 
degree of simileirity between the scene -change frame and 
the image that has been designated at said designation 
step; 

wherein said scene extraction step extracts the 
scene corresponding to said image based upon results of 
calculation performed at said calculation step. 

12. The method according to claim 10, further 
comprising an output step of outputting scene 
information relating to scenes that have been extracted 
at said scene extraction step. 

13. The method according to claim 12, wherein said 
output means edits scenes that have been extracted at 
said scene extraction step and combines these extracted 
scenes into a single moving picture. 

14. The method according to claim 10, wherein said 
designating step designates a pattern image that 
corresponds to any of a leading, intermediate or final 
frame of a scene that is the object of a search. 

15. The method according to claim 10, wherein said 
designating step is capable of designating a number of 
scenes to be extracted. 

16. The method according to claim 10, wherein said 
designating step is capable of designating the time of a 
scene to be extracted. 

17. The method according to claim 10, wherein said 
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designating step is capable of designating a number of 
scenes to be extracted, with regard to frames prior to 
and with regard to frames on and after a frame 
corresponding to the pattern image. 

18. The method according to claim 10, wherein said 
designating step is capable of designating time of a 
scene to be extracted, with regard to frames prior to 
and with regard to frames on and after a frame 
corresponding to the pattern image. 

19. A computer-readable memory storing program code of 
image processing for processing a moving picture, the 
memory including: 

program code of a frame extraction step of 
extracting frames constituting an entered moving 
picture; 

program code of a discrimination step of 
discriminating a scene change by comparing frames 
extracted at said frame extraction step; 

program code of a storage step of storing scene- 
change information relating to the scene change 
discriminated at said discrimination step; 

program code of a designating step of designating 
an image that corresponds to a scene that is the object 
of a search; 

program code of a comparison step of comparing a 
scene-change frame, which is obtained by referring to 
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the scene-change information that has been stored at 
said storage step, and the image that has been 
designated at said designation step; and 

program code of a scene extraction step of 
extracting a scene that corresponds to the image based 
upon result of the comparison performed at said 
comparison step. 



ABSTRACT OF THE DISCLOSURE 
Frames constituting an input moving picture are 
extracted by a frame extraction unit, the extracted 
frames are compared and scene changes in the moving 
5 picture are discriminated by a scene-change 

discrimination unit. Scene-change information relating 
to specific scene changes is stored in a scene-change 
information storage unit. An image corresponding to a 
scene that is the object of a search is designated from 

10 a pattern-image designation unit. The designated image 
is compared with scene-change frames obtained by 
referring to the scene-change information that has been 
stored in the scene-change storage unit, and a scene 
corresponding to the image is extracted by a scene 

15 extraction unit based upon results of the comparison. 
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FIG. 6 
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